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FIELD OF THE INVENTIO ^^^ ^^ . 0 ^. 
The present invention relates to imaging generally and more particularly to post 
acquisition image processing generally. 



BACKGROUND OF THE INVENTION 

Post acquisition image processing is well known in the literature. Publications 
which describe the general state of the art in post acquisition image processing are : Lim, J.S. 
"Two dimensional signal and image processing". Englewood Cliffs, NT Prentice Hall. (1990); 
Russ, J.C. "Image processing handbook". CRC Press. (1992); Pratt, W.K. "Digital image 
processing". NY John Wiley & Sons, Inc. (1991); Rosenfeld, A. and Kak, A.C. "Digital picture 
processing". Academic Press. (1976); Castleman, K.R. "Digital Image Processing". Prentice- 
Hall Inc. Englewood Cliffs, New Jersey. (1979). 

Imaging which provides information relating to refractive characteristics in a 
imaged volume is known for extremely limited applications. In microscopy, Smith, Nomarski 
and Differential Interference Contrast (DIC) imaging is known and is described in the 
following publications: Nomarski, G. "Microinterferometre differential a ondes polarisees". J. 
Phys. Radium 16:9S-11S (1955); Lang, W. "Differential-Interferenz-Miikroskopie". Carl- 
Zeiss, Oberkochen (1975); Inoui, S. and Spring, K.S. "Video Microscopy: the fundamentals". 
2nd edition. Plenum Press, NY. (1997). Tanford, C. "Physical chemistry of macromolecules". 
John Wiley NY. (1961). Appendix C describes classical Rayleigh interference methods, Philpot 
and Svenson methods based on schlieren image, Lamm method of line displacement, and Gouy 
interference method all developed for determination of one dimensional refractive index 
variations. 

Computer analysis of DIC imaging is not readily achieved. Known instances are 
described in the following publications: Allen, R.D., Allen, N.S. and Travis, J.L. "Video- 
enhanced contrast, differential interference contrast (AVEC-DIC) microscopy: a new method 
capable of analyzing microtubule related motility in the reticulopodial network of Allogromia 
laticollaria." Cell Motility 1: 291-302 (1981); Cogswell, C.J. and Sheppard, CJ.R. "Confocal 


differential interference -contrast (DIC) microscopy: including a „^eoreticd aii^ysis of 
conventional and confocal DIC imaging". J. Microsc. 165:81-101 (1992); Gelles, J., Schnapp, 
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B.J. and Sheetz, MP. "Tracking kinesin-driven movements with nanometre-scale precision", 
Nature 331:450-453 (1988); Hdusler, G. and Kvrner, E. "Imaging with expanded depth of 
focus". Zeiss Inform. 29: 9-13 (1987); Preza, C, Snyder, D.L. and Conchello, J-A. "Image 
reconstruction for three-dimensional transmitted light DIC microscopy". SPEE 2984:220-231 
(1997);.Schorrnann, T. and Jovin, T.M. "Contrast enhancement and depth perception in three- 
dimensional representations of differential interference contrast and confocal scanning laser 
microscope images". J. Microsc. 166:155-168 (1992). 

Computerized ray tracing between discrete refractive and reflective surfaces is 
extremely well developed, but is not well known in the environment of non-homogeneous 
indices of refraction. This area is described in the following publications: Hecht E. and Zajac 
A. "Optics" 2nd ed. Addison- Wesley Reading MA (1997); Jenkins, F.A, and White, H.E. 
"Fundamentals of optics". McGraw-Hill, NY (1950) ch.8: Ray Tracing. 

Calculation of point spread functions (PSF) is extremely well known as 
described in the following publication: Born M. and Wolf E. "Principles of Optics" Pergamon 
London (1959);Goodman J.W. "Statistical Optics" John Wiley & Sons NY (1985);Hecht E. 
and Zajac A. "Optics" 2nd ed. Addison-Wesley Reading MA. (1997); Gibson S.F. and Lanni F. 
"Diffraction by circular aperture as a model for three-dimensional optical microscopy". Opt. 
Soc. Am. A 6:1357-1367 (1989); Gibson S.F. and Lanni F. "Modeling aberrations due to 
mismatched layers for 3-D microscopy" SPIE optics in complex systems 1319:470-471 (1990); 
Gibson S.F. and Lanni F. "Experimental test of an analytical model of aberration in an oil- 
immersion objective lens used in three-dimensional light microscopy". J. Opt. Soc. Am. A 
3:1601-1613 (1991). 

Deconvolution of three dimensional microscopic images having location 
independent PSF is well known and is described in the following publications, some of them 
authored by some of the present inventors: Jansson, P. A. ed. "Deconvolution of images and 
spectra". Academic Press NY (1997); Agard, D A. and Sedat, J.W.. "Three-dimensional 
architecture of a polytene nucleus". Nature 302:676-681 (1984); Agard, D.A., Hiraoka, Y., 
Shaw, P. and Sedat, J.W. "Fluorescence microscopy in three dimensions". Methods in Cell 
Biology 30: 353-377 (1989); Castleman, K.R. "Digital Image Processing". Prentice-Hall Inc. 
Englewood Cliffs, New Jersey (1979). Correction of telescopic images by the use of suitably 
distorted mirrors and deconvolution of two dimensional telescope images having location 
pendeW = PSF^are^=de 


Hanisch, R.J., Mo, J. and White, R. "Comparative results with massively "parallel spatially- 


variant maximum likelihood image restoration". Bui Am Astr. Soc 27:924-929 (1995); Boden, 
A.F., Reeding, DC, Hanisch, RJ. and Mo, J. "Massively parallel spatially-variant maximum 
likelihood restoration of Hubble space telescope imagery". J Opt Soc Am A 13: 1537-1545 
(1996); Jansson, P.A. ed. "Deconvolution of images and spectra". Academic Press NY (1997); 
Tyson R.K. "Principles of Adaptive Optics" Academic Press NY (1991). Reconstruction of 
blurred images from point objects is described in the following publications: Carrington, W.A., 
Lynch, R.M., Moore, D.W., Isenberg, G., Fogarty, K.E. and Fay, F.S. "Superresolution three- 
dimensional images of fluorescence in cells with minimal light exposure". Science 268:1483- 
1487 (1995); Femino, A.M., Fay, F.S., Fogarty, K., and Singer, R.H. "Visualization of single 
RNA transcripts in situ". Science 280:585-590 (1998). 

SUMMARY OF THE INVENTION 
The present invention seeks to provide improved apparatus and techniques for 
j post acquisition image processing. 

I There is thus provided in accordance with a preferred embodiment of the 

\ present invention apparatus for computational adaptive imaging including an image information 
S acquirer providing information relating to the refractive characteristics in a three-dimensional 
3 imaged volume, a ray tracer, utilizing the information relating to the refractive characteristics 
I to trace a multiplicity of rays from a multiplicity of locations in the three-dimensional imaged 
3 volume through the three-dimensional imaged volume, thereby providing a location dependent 
I point spread fimction and a deconvolver, utilizing the location dependent point spread 
function, to provide an output image corrected for distortions due to variations in the 
refractive characteristics in the three-dimensional imaged volume. 

Preferably, the image information acquirer acquires at least two three- 
dimensional images of a three-dimensional imaged volume, at least one of the two three- 
dimensional images containing the information relating to the refractive characteristics in a 
three-dimensional imaged volume. 

When the refractive characteristics are extractable from the image to be 
corrected for distortions, or are known independently, only one three-dimensional image need 
be acquired. 

The acquirer may obtain refractive index information from DIG, for example 

_____ a . p, fi niliv .crr n rr Xm ex am pl e in PNA-n.^nciatcd qtnins wherein ^ 

IrorfrphBse micro sco p y o r from flou iesecitcc— itji-c*a*»t«^ *» 


the stain intensity is proportional to the refractive index increment. 
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Refractive index mapping may be applied to samples whose refractive index is 
known. For example this may apply to microchip wafer structures, whose geometry is known. 

In accordance with a preferred embodiment of the present invention, the image 
acquirer acquires at least three three-dimensional images of the three-dimensional imaged 
volume. 

Preferably, the image acquirer acquires a plurality of three-dimensional images 
of the three-dimensional imaged volume, each the image having a discrete wavelength band. 

Alternatively, the image acquirer acquires a multiplicity of three-dimensional 
images of the three-dimensional imaged volume, each the image having a wavelength band 
which is part of a continuum represented by the wavelength bands of the multiplicity of three- 
dimensional images. 

In accordance with a preferred embodiment of the present invention, the ray 
O tracer and the deconvolver utilize the information relating to the refractive characteristics in a 
Hj three-dimensional imaged volume obtained from one of the three-dimensional images to 

! : ! 

% correct at least another one of the three-dimensional image or itself. 

J The acquirer may obtain refractive index information from DIC, or from phase 

IT microscopy or from flourescence -for example in DNA associated stains wherein the stain 
p intensity is proportional to the refractive index increment. 

J: Refractive index mapping may be applied to samples whose refractive index is 

S J known. For example this may apply to microchip wafer structures, whose geometry is known. 
2 According to one embodiment of the present invention, the three-dimensional 

images are electromagnetic energy images. Preferably, the three-dimensional images are 

infrared images. 

Alternatively, the three-dimensional images are non-electromagnetic images. 

In accordance with a preferred embodiment of the present invention, the image 
acquirer receives digital image data from a digital image source and derives therefrom the 
information relating to the refractive characteristics in a three-dimensional imaged volume. 

Preferably, the ray tracer and the deconvolver operate repeatedly over time to 
provide a multiplicity of output images, each corrected for distortions due to variations in the 
refractive characteristics in the three-dimensional imaged volume. Such a deconvolution 
process can be iteratively applied to the whole process to improve the estimation of the 
r eft act i v e ind ex . — 
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In accordance with one embodiment of the present invention, the output image 
is an acoustic image and the refractive characteristics are characteristics of a material which the 
passage of acoustic energy therethrough. 

In accordance with an alternative embodiment of the present invention, the 
output image is an electromagnetic image and the refractive characteristics are characteristics 
of a material which the passage of electromagnetic energy therethrough. 

There is also provided in accordance with a preferred embodiment of the 
present invention a method for computational adaptive imaging including the steps of: 

providing information relating to the refractive characteristics in a three- 
dimensional imaged volume; 

ray tracing, utilizing the information relating to the refractive characteristics, a 
multiplicity of rays from a multiplicity of locations in the three-dimensional imaged volume 
through the three-dimensional imaged volume, thereby providing a location dependent point 

spread function; and 

deconvolving, utilizing the location dependent point spread function, thereby 
providing an output image corrected for distortions due to variations in the refractive 
characteristics in the three-dimensional imaged volume. Such a deconvolution process can be 
iteratively applied to the whole process to improve the estimation of the refractive index. 

Preferably, the step of providing information includes acquiring at least two 
three-dimensional images of a three-dimensional imaged volume, at least one of the two three- 
dimensional images containing the information relating to the refractive characteristics in a 
three-dimensional imaged volume. When the refractive characteristics are extractable 

from the image to be corrected for distortions, or are known independently, only one three- 
dimensional image need be acquired. 

In accordance with a preferred embodiment of the present invention, the step of 
providing information includes acquiring at least three three-dimensional images of a three- 
dimensional imaged volume. 

Preferably, the step of providing information includes acquiring a plurality of 
three-dimensional images of the three-dimensional imaged volume, each image having a 
discrete wavelength band. 

In accordance with a preferred embodiment of the present invention, the step of 
^ F0V 4^^^i ?Pa tinn^ ncludes ac qu i rin g a mult i pl i city of t h ree-d i mens i onal images of the 
three-dimensional imaged volume, each the image having a wavelength band" which is part of a 


ru 
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continuum represented by the wavelength bands of the multiplicity of three-dimensional 
images. 

In accordance with one embodiment of the present invention, the three- 
dimensional images are electromagnetic energy images. Preferably, the three-dimensional 
images are infrared images. 

In accordance with an alternative embodiment of the present invention, the 
three-dimensional images are non-electromagnetic images. 

The refractive index may be in any medium and the imaging method may be for 
a generalised method for inhomogeneous media that may distort the image. 

Preferably, the step of providing includes receiving digital image data from a 
digital image source and deriving therefrom the information relating to the refractive 
characteristics along a multiplicity of light paths in a three-dimensional imaged volume. 

In accordance with a preferred embodiment of the present invention, the steps 
of providing information, ray tracing and deconvoluting operate repeatedly over time to 
provide a multiplicity of output images, each corrected for distortions due to variations in the 
refractive characteristics in the three-dimensional imaged volume. Such a deconvolution 
process may be iteratively applied to the whole process to improve the estimation for the 
refractive index map. 

There is also provided in accordance with a preferred embodiment of the 
present invention apparatus for utilizing differential interference contrast images to provide 
three-dimensional refractive index information including a line integrator operating on 
differential interference contrast images displaying a directional derivative of refractive index of 
an object to invert the directional derivative thereof, thereby providing a plurality of two- 
dimensional representations of the refractive index of the object. Alternatively the three- 
dimensional refractive index map can be obtained from phase microscopy, or from fluorescence 
where the staining is proportional to the refractive index increment. 

There is additionally provided in accordance with a preferred embodiment of 
the present invention apparatus for utilizing differential interference contrast images to provide 
three-dimensional refractive index information and also including a deconvolver performing 
deconvolution of the plurality of two-dimensional representations of the refractive index of the 
object, thereby reducing out-of-focus contributions to the two-dimensional representations of 
^e=fefractt ve4ndex-of-thc obj e ct. = 


There is further provided in accordance with a preferred embodiment of the 
present invention a method for utilizing differential interference contrast images to provide 
three-dimensional refractive index information including performing line integration on 
differential interference contrast images displaying a directional derivative of refractive index of 
an object to invert the directional derivative thereof, thereby providing a plurality of two- 
dimensional representations of the refractive index of the object. Alternatively the three- 
dimensional refractive index map can be obtained from phase microscopy, or from 
flourexcence where the staining is proportional to the refractive index increment. 

Additionally in accordance with a preferred embodiment of the present 
invention there is provided a method for utilizing differential interference contrast images to 
provide three-dimensional refractive index information and also including performing 
deconvolution of the plurality of two-dimensional representations of the refractive index of the 
object, thereby reducing out-of-focus contributions to the two-dimensional representations of 
the refractive index of the object. Again, the three-dimensional refractive index map can be 
obtained from phase microscopy, or from flourexcence where the staining is proportional to 
the refractive index increment. 

Further in accordance with a preferred embodiment of the present invention 
there is provided apparatus for ray tracing through a medium having multiple variations in 

refractive index including: 

a computer employing an analytically determined path of a ray through the 
multiplicity of locations in the medium, for a plurality of rays impinging thereon in different 
directions, by utilizing known local variation of the refractive index at a multiplicity of 

locations in the medium. 

There is additionally provided in accordance with a preferred embodiment of 
the present invention a method of ray tracing through a medium having multiple variations in 
refractive index including: 

determining local variation of the refractive index at a multiplicity of locations 

in the medium; 

analytically determining the path of a ray through the multiplicity of locations in 
the medium, for a plurality of rays impinging thereon in different directions. The ray tracing 
may also include the computation of absorptions, reflections and scattering of rays and their 
contributions to^t he4magtfM £ proc e ss. 
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Still further in accordance with a preferred embodiment of the present invention 
there is provided apparatus for confocal microscopy including: 

a ray tracer, employing known variations of the refractive index in a three- 
dimensional sample for determining the paths of a multiplicity of rays emerging from at least 
one point in the sample and passing through the sample, thereby determining an aberrated 
wavefront for each the point; and 

an adaptive optics controller utilizing the aberrated wavefront to control an 
adaptive optical element in a confocal microscope, thereby to correct aberrations resulting 
from the variations in the refractive index. 

There is additionally provided in accordance with a preferred embodiment of 
the present invention a method for confocal microscopy including: 

determining variations of the refractive index in a three-dimensional sample; 

determining the paths of a multiplicity of rays emerging from at least one point 
in the sample and passing through the sample, thereby determining an aberrated wavefront for 
each the point; and 

utilizing the aberrated wavefront to control an adaptive optical element in a 
confocal microscope, thereby to correct aberrationresulting from the variations in the refractive 
index. 

There is additionally provided in accordance with a preferred embodiment of 
the present invention a method for adding (computationally or physically) in the imaging path a 
three-dimensional medium (anti-sample) with refractive properties that correct for the 
distortions of the three-dimensional sample. 


BRIEF DESCRIPTION OF THE DRAWINGS 
The present invention will be understood and appreciated more fully from the 

following detailed description, taken in conjunction with the drawings in which: 

Figs. 1A and IB are respectively representations of three dimensional 

microscopic images of an object according to the prior art and in accordance with the present 

invention; 

Fig. 2 is a simplified partially pictorial, partially block diagram illustration of 
apparatus for computational adaptive imaging constructed and operative in accordance with a 
refCT^d^emtyodim e r U of the present invention; — 
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Fig. 3 is a simplified flow chart of the operation of the apparatus for 
computational adaptive imaging of Fig. 2; 

Fig. 4 is a simplified diagram of line integration functionality which is carried 
out by the apparatus of Fig. 2; 

Fig. 5 is a simplified diagram of ray tracing functionality which is carried out by 
the apparatus of Fig. 2; 

Fig. 6 is a simplified diagram of wavefront integration functionality which is 
carried out by the apparatus of Fig. 2; and 

Fig. 7 is a simplified flow chart illustrating deconvolution functionality which is 
carried out by the apparatus of Fig. 2. 


BRIEF DESCRIPTION OF APPENDICES 
Microfiche appendix A is a hexadecimal listing of a program which converts 
DIC images to a three dimensional map of refraction indices of a corresponding imaged object 
by line integration; 

Microfiche appendix B is a hexadecimal listing of a program which calculates 
location dependent PSF by ray tracing through the imaged object, taking into account the three 
dimensional map of refraction indices thereof, calculating a distorted wavefront and computing 
therefrom a diffraction integral, thereby to provide the PSF; 

Microfiche appendix C is a hexadecimal listing of a program which 
deconvolutes the location dependent PSF from a distorted image, thus providing a corrected 
image of the imaged object. 

DETAILED DESCRIPTION OF THE INVENTION 
Reference is now made to Figs. 1A and IB, which are respectively 
representations of three dimensional microscopic images of an object according to the prior art 
and in accordance with the present invention. 

It is known in the prior art to image a point source or a collection of point 
sources in an ideal sample displaying no substantial variation in refractive index. When an ideal 
optical system is focussed at slightly different depths in the sample, a three dimensional image 
appears, including a plurality of two-dimensional diffraction limited images designated by 
^Roman numcralsU,JLI, TT I, T V a n d V in Fig 1 A Th e prior art, described, inter alia in t fas= 


aforesaid Agard et al 1989 reference is capable" of deconvolving the plurality of two- 
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dimensional images I - V to a deconvolved three dimensional image, including a single in 
focus two-dimensional image VI, which contains substantially all of the image intensity from 
images I - V and a plurality of out of focus two-dimensional images which substantially contain 
no image intensity. 

The prior art, however, does not know how to deal adequately with samples 
which display substantial variation in refractive index. 

The present invention provides a solution to this problem, as illustrated in Fig. 
IB which shows images of a non-ideal sample, such as a biological sample, displaying 
substantial variation in refractive index. When an ide^l optical system is focussed at slightly 
different depths in the non-ideal sample, a three-dimensional image appears, including a 
plurality of two-dimensional diffraction limited images designated by Roman numerals I, II, III, 
IV and V in Fig. IB. In contrast to the images appearing in Fig. 1 A, it is seen that these images 
are extremely non-uniformly distorted due to variations in the refractive index at various 
locations in the sample. 

The present invention, as will be described hereinbelow, is capable of 
deconvoluting the plurality of two-dimensional images I - V in Fig. IB, notwithstanding their 
great distortions due to variations in the refractive index at various locations in the sample, to a 
substantially non-distorted, deconvoluted three-dimensional image, including a single in-focus 
two-dimensional image VI, which contains substantially all of the image intensity from images I 
- V and a plurality of out-of- focus two-dimensional images which substantially contain no 
image intensity. 

Reference is now made to Fig. 2, which is a simplified partially pictorial, 
partially block diagram illustration of apparatus for computational adaptive imaging 
constructed and operative in accordance with a preferred embodiment of the present invention. 
The apparatus of Fig. 2 typically includes a conventional computer-controlled optical 
microscope 10, such as a microscope manufactured by Carl Zeiss and sold under the name 
Axioskop. Associated with microscope 10 there are preferably provided computer controlled 
filter wheels 12 and 14. Filter wheel 12 is disposed downstream of an excitation light source 
16, such as a mercury arc lamp, which provides epi-illumination via a computer-controlled 
shutter 18 and filter wheel 12 to a dichroic mirror 20 within the microscope 10. 

Trans-illumination is provided to a sample 22, typically mounted between two 
^li de s^4=aiKi^H>y" me an s-o^ a c omputer-controlled s hutter 30. 

Depending on which of shutters 18 and 30 is open, the sample "receives "tTan^illufmnation or 
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epi-illumination so as to provide either a DIC image or a fluorescent image respectively. It is 
noted that DIC images conventionally require additional optical elements in the imaging light 
path, which are omitted from Fig. 2 for the sake of conciseness and clarity. 

In accordance with a preferred embodiment of the present invention, the DIC or 
fluorescent image is received via computer-controlled filter wheel 14 by a digital CCD sensor 
32, such as a Photometries PXL. Preferably a polarizer is included in one of the windows of 
filter wheel 14 and is employed for DIC imaging. 

In accordance with a preferred embodiment of the present invention, the output 
of the digital CCD sensor 32 is supplied to a computer 34, such as a Silicon Graphics 02 
workstation which performs the following functionality: 

ray tracing, utilizing information relating to the refractive characteristics in the 
sample, a multiplicity of rays from a multiplicity of locations in the sample through the sample, 
thereby providing a location dependent point spread function; and 

deconvoluting, utilizing the location dependent point spread function, thereby 
providing an output image corrected for distortions due to variations in the refractive 
□ characteristics in the three-dimensional imaged volume. 

n] Preferably wavefront integration is performed following the ray tracing and 

^ prior to the deconvoluting. 

[IS In accordance with a preferred embodiment of the present invention, the 

\p ; information relating to the refractive characteristics of the sample is provided by line 

integration of the DIC image. 
H= Reference is made to Fig. 3, which is a simplified flow chart of the operation of 

Si the apparatus for computational adaptive imaging of Fig. 2. As illustrated in Fig. 3, three 
^ dimensional images from the microscope 10 and CCD 32, typically include a plurality of 
observed fluorescent 3D images 40 and a differential interference contrast (DIC) 3D image 42. 

In accordance with a preferred embodiment of the present invention, line 
integration 44 is performed on the DIC image, preferably using the software exemplified in 
Appendix A. The line integration preferably provides a 3D refractive index map 46. 

In accordance with a preferred embodiment of the present invention, ray tracing 
48 is performed on the 3D refractive index map 46 and the resulting wavefront is subject to 
wavefront integration 50, thus providing a location dependent point spread function (PSF) 52. 
Preferably the ray tracing 48 and the wavefront integrat ion 50 is performed by using the 
software exemplified in Appendix B. - 



In accordance with a preferred embodiment of the present invention, the 
location dependent PSF is utilized in a deconvolution step 54 which is applied to the observed 
fluorescent 3D image 40. Preferably the deconvolution 54 is performed by using the software 
exemplified in Appendix C, 

In accordance a preferred embodiment of the invention, the deconvolution 
produces a corrected image, such as the image illustrated at reference VI in Fig. IB. 

Reference is now made to Fig. 4, which is a simplified diagram of line 
integration functionality which is carried out by the apparatus of Fig. 2, preferably utilizing the 
software exemplified in Appendix A. Fig. 4 illustrates a sample, such as a sphere 60 of uniform 
index of refraction, which differs from that of surrounding media 62. Sample 60 may be taken 
to represent a nucleus, while the surrounding media may be taken to represent cytoplasm. A 
scan line 64 illustrates the index of refraction as a function of location along line 65. The 
characteristic PSF of the microscope 10 operating in the DIC imaging mode is illustrated at 
3 reference numeral 66. The PSF is also illustrated by trace 68. 

m The output image of sample 60 of the CCD sensor 32 operating in a DIC mode 

is illustrated at reference numeral 70 and also by trace 72. Line integration of the output image, 
y= preferably utilizing the software exemplified in Appendix A, produces an image 74 which 
y] accurately represents the original sample, seen at reference numerals 60 and 62. 
13 It is appreciated that line integration of DIC images has two complex 

p contributions in reality. One is the accumulation of noise in the integration and the second is 
*M the slow contribution of low-intensity out-of-focus contributions. The problem of noise is 
preferably alleviated by introducing a decay function in the line integration. This decay feature 
appears in Appendix A. The out-of-focus contribution can be reduced significantly by 
deconvoluting the integrated DIC images. This feature also appears in Appendix A. 

Reference is now made to Fig. 5, which is a simplified diagram of ray tracing 
functionality which is carried out by the apparatus of Fig. 2. As seen in Fig. 5, it is seen that an 
isotropic fan of rays is drawn from every point in an imaged volume, e.g. the sample. These 
rays are ray traced, preferably using the software embodied in Appendix B, such that their 
paths and phases are modified in accordance with variations in the refractive index of the 
sample. The result of the ray tracing is an aberrated wavefront for every point in the imaged 
volume. 

Reference is now made to Fig. 6. which is a simplified diagram of prior art 
wavefront integration functionality which is carried out by the apparatus of Figr 2 preferably 
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using the software embodied in Appendix B. As seen in Fig. 6, for every point in the vicinity of 
a focus RO, an interference integral is taken over the wavefront. The integrations for all of the 
points in the vicinity of the focus RO define the three-dimensional point spread function, as 
described inter alia in the aforesaid prior art reference to Goodman. 

Reference is now made to Fig. 7, which is a simplified flow chart illustrating 
deconvolution functionality which is carried out by the apparatus of Fig. 2, preferably utilizing 
the software embodied in Appendix C. As seen in Fig. 7, the observed image 80, such as the 
observed fluorescent 3D image 40 (Fig. 3) is deconvoluted using the point spread function 
(PSF) calculated as described hereinabove with reference to Figs. 5 and 6. It is appreciated that 
the location dependent PSF is preferably a multi-resolution PSF. This feature appears in the 
software of Appendix C. This calculation is preferably performed in an iterative process which 
endeavors to provide progressively better approximations of the imaged sample. 

This iterative process typically proceeds by initially defining a guessed object 
82, which typically is identical to the observed image. The guessed object is convoluted with a 
O location dependent point spread function 84, such as that derived by ray tracing and wavefront 
5 s ! integration (Fig. 3) to produce a blurred image 86. This blurred image is compared with the 
jjj observed image and a correction based on the comparison is iteratively applied to the guessed 
fU object. Following multiple iterations, the guessed object constitutes the corrected 3D image of 
IS the sample in accordance with a preferred embodiment of the present invention. 
i_ A preferred method for constructing a system operative in accordance with a 

Ms preferred embodiment of the present invention is now described with reference to Appendices 
Sj A, B, and C: 

^ a) Provide a SILICON GRAPHICS workstation, such as model 02, with 128 

megabytes of RAM memory running the IRIX Release 6.3 IP32 operating system; 

b) Install the DELTAVISION optical sectioning microscope system, commercially 
available from Applied Precision Inc., 1024 12th Ave NW, Issaquah WA 98027-8929, USA; 

c) Create three binary files from the hexadecimal listings of Appendices A, B and 
C; 

d) Uncompress the three binary files using GZIP version 1.1.2 into three 
executable files named "pardic," "dictrace," and "decon_fuli"; 

e) Provide at least one 3D DIC image and one 3D fluorescent image using the 
DE LIAV1 SIQN s yst e m; 
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f) Run the pardic executable file to perform the following: 

Read one optical section from the 3D DIC image; 

Use the 'F submenu to filter the image from slowly varying 

backgrounds; 

Use the line integration submenu IF to convert the 3D DIC image to a 
3D refractive index map; 

For 'filling up* the effect of decay in the line integration use DIC- 
deconvolution submenu W. To do this, first define the DIC point spread function using the 
kernel submenu 'K'; 

For an improved index map, read three consecutive optical DIC sections 
and use the 'Z' submenu to subtract out-of-focus contributions using nearest neighbor 
approximation of the three-dimensional deconvolution; 

Output the refractive index 3D map section by section; 

g) Run the dictrace executable file to perform the following: 

Read in the refractive index 3D map. Simulated maps including different 
gradient functions are available as alternative inputs; 

Create a fan of rays originating from a point and homogeneously 
spanning a cone with a defined solid angle. The opening angle and angular increment can be 
defined. Typically 200,000 rays are launched into a solid angle cone of 60 degrees; 

Trace the rays through the refractive index 3D mapped volume, and 
calculate the position of intersection and the phase of each of the rays at the entrance pupil 
wavefront sphere; 

Transform the converging wavefront in the entrance pupil of the optical 
system to a diverging wavefront at the exit pupil, and calculate the 3D point spread function 
corresponding to this wavefront using the Kirchhoff interference integral submenu; 

Output the 3D point spread function for the calculated location of the 
source point of the fan of rays; 

h) Run the decon_fulI executable file to perform the following: 

Input the location-dependent 3D point spread function and the fluorescent 
image for deconvoluting, and run the iterative deconvolution. The program outputs the 
deconvoluted improved 3D image. 

Additional technical descriptions-of a preferred embodiment of the invention 
appear in the following article by the inventor Z. Kam: "Microscopic Differential~Interference 
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Contrast Image Processing by Line Integration (LID) and Deconvolution" Bioimaging 6 (4): 
166-176 (1998), and which is incorporated herewith. 

It will be appreciated by persons skilled in the art that the present invention is 
not limited to what has been particularly shown and described hereinabove. Rather the scope of 
the present invention includes both combinations and subcombinations of the various features 
described hereinabove as well as modifications and additions thereto which would occur to a 
person of skill in the art upon reading the foregoing description and which are not in the prior 
art. 
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CLAIMS 

We claim: 

1. Apparatus for computational adaptive imaging including: 

an image information acquirer providing information relating to the refractive 
characteristics in a three-dimensional imaged volume; 

a ray tracer, utilizing the information relating to the refractive characteristics to 
trace a multiplicity of rays from a multiplicity of locations in the three-dimensional imaged 
volume through the three-dimensional imaged volume, thereby providing a location dependent 
point spread function; and 

a deconvolves utilizing the location dependent point spread function, to 
provide an output image corrected for distortions due to variations in the refractive 
characteristics in the three-dimensional imaged volume. 

2. Apparatus according to claim 1 wherein the acquirer is adapted to obtain 
refractive index information from DIC, for example from phase microscopy or from 
flourescence. 

3. Apparatus according to claim 1 wherein the acquirer is adapted to utilise 
previously determined refractive characteristics. 

4. Apparatus for computational adaptive imaging according to claim 1 and wherein 
the image information acquirer acquires at least two three-dimensional images of a three- 
dimensional imaged volume, at least one of the two three-dimensional images containing the 
information relating to the refractive characteristics in a three-dimensional imaged volume. 

5. Apparatus according to claim 4 and wherein the image acquirer acquires at least 
three three-dimensional images of the three-dimensional imaged volume. 

6. Apparatus according to claim 4 and wherein the image acquirer acquires a 
plurality of three-dimensional images of the three-dimensional imaged volume, each the image 
having a discrete wavelength band. 


7 Apparatus according to claim 4 and wherein the image acquirer acquires a 
multiplicity of three-dimensional images of the three-dimensional imaged volume, each the 
image having a wavelength band which is part of a continuum represented by the wavelength 
bands of the multiplicity of three-dimensional images. 

8 Apparatus according to claim 4 wherein the image acquirer acquires a single 
three-dimensional image of the three-dimensional image volume. 

9 Apparatus according to' claim 4 and wherein the ray tracer and the 
deconvolver utilize the information relating to the refractive characteristics in a three- 
dimensional imaged volume obtained from one of the three-dimensional images to correct at 
least another one of the three-dimensional images. 

10 Apparatus according to claim 9 wherein the ray tracer includes effects on the 
image of absorptions, reflections and scattering in the sample. 

n Apparatus according to claim l, and wherein the three-dimensional images are 

electromagnetic energy images. 

12 Apparatus according to claim il and wherein the three-dimensional images are 
infrared images. 

13 Apparatus according to claim 1, and wherein the three-dimensional images are 
non-electromagnetic images. 

14 Apparatus according to claim 1, and wherein the image acquirer receives digital 
image data from a digital image source and derives therefrom the information relating to the 
refractive characteristics in a three-dimensional imaged volume. 

15 Apparatus according to claim 1, and wherein the image acquirer, the ray tracer 
and the deconvolver operate repeatedly over time to provide a multiplicity of output images, 
each corrected for distortions due to variations in the refractive characteristics in the three- 
dimensionai imaged volume. ' ~ — 
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16. Apparatus according to claim l, and wherein the output image is an acoustic 
image and the refractive characteristics are characteristics of a material which the passage of 
acoustic energy therethrough. 

17. Apparatus according to claim 1, and wherein the output image is an 
electromagnetic image and the refractive characteristics are characteristics of a material which 
the passage of electromagnetic energy therethrough. 

18. A method for computational adaptive imaging including the steps of: 
providing information relating to the refractive characteristics in a three- 
dimensional imaged volume; 

ray tracing, utilizing the information relating to the refractive characteristics, a 
multiplicity of rays from a multiplicity of locations in the three-dimensional imaged volume 
through the three-dimensional imaged volume, thereby providing a location dependent point 
spread function; and 

deconvolving, utilizing the location dependent point spread function, thereby 
providing an output image corrected for distortions due to variations in the refractive 
characteristics in the three-dimensional imaged volume. 

19. A method according to claim 18, for adding in the imaging path a three- 
dimensional medium (anti-sample) with refractive properties that correct for the distortions of 
the three-dimensional sample. 

20. A method for computational adaptive imaging according to claim 18 and 
wherein the step of providing information includes acquiring at least two three-dimensional 
images of a three-dimensional imaged volume, at least one of the two three-dimensional images 
containing the information relating to the refractive characteristics in a three-dimensional 
imaged volume. 

21. A method according to claim 20 and wherein the step of providing information 
includes- a c quirin g at least three three-dimensional images of a three-dimension? ! imaged 


22. A method according to claim 20 and wherein the step of providing information 
includes acquiring a plurality of three-dimensional images of the three-dimensional imaged 
volume, each the image having a discrete wavelength band. 

23. A method according to claim 20 and wherein the step of providing information 
includes acquiring a multiplicity of three-dimensional images of the three-dimensional imaged 
volume, each the image having a wavelength band which is part of a continuum represented by 
the wavelength bands of the multiplicity of three-dimensional images. 

24. A method according to claim 18, and wherein the three-dimensional images are 
electromagnetic energy images. 

25. A method according to claim 24 and wherein the three-dimensional images are 
infrared images. 

26. A method according to claim 18, and wherein the three-dimensional images are 
non-electromagnetic images. 

27. A method according to claim 18, and wherein the step of providing includes 
receiving digital image data from a digital image source and deriving therefrom the information 
relating to the refractive characteristics along a multiplicity of light paths in a three-dimensional 
imaged volume. 

28. A method according to claim 18, and wherein the steps of providing 
information, ray tracing and deconvolving operate repeatedly over time to provide a 
multiplicity of output images, each corrected for distortions due to variations in the refractive 
characteristics in the three-dimensional imaged volume. 

29. A method according to claim 28 wherein the refractive characteristics are 
estimated. 


30. Apparatus for utilizing differential interference contrast images to provide 
three-dimensional refractive index information including a line integrator operating on 
differentia] interference contrast images displaying a directional derivative of refractive index of 
an object to invert the directional derivative thereof, thereby providing a plurality of two- 
dimensional representations of the refractive index of the object. 

31. Apparatus for utilizing differential interference contrast images to provide 
three-dimensional refractive index information according to claim 30 and also including a 
deconvolver performing deconvolution of the plurality of two-dimensional representations of 
the refractive index of the object, thereby reducing out-of-focus contributions to the two- 
dimensional representations of the refractive index of the object. 

32. A method for utilizing differential interference contrast images to provide three- 
dimensional refractive index information including performing line integration on differential 
interference contrast images displaying a directional derivative of refractive index of an object 
to invert the directional derivative thereof, thereby providing a plurality of two-dimensional 
representations of the refractive index of the object. 

33. A method for utilizing differential interference contrast images to provide three- 
dimensional refractive index information according to claim 32 and also including performing 
deconvolution of the plurality of two-dimensional representations of the refractive index of the 
object, thereby reducing out-of-focus contributions to the two-dimensional representations of 
the refractive index of the object. 

34. Apparatus for ray tracing through a medium having multiple variations in 
refractive index including: 

a computer employing an analytically determined path of a ray through the 
multiplicity of locations in the medium, for a plurality of rays impinging thereon in different 
directions, by utilizing known local variation of the refractive index at a multiplicity of 
locations in the medium. 

35. Armethod^ofjay tracing throughAmedium having multiple variations irefractive- 

index including: 
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determining local variation of the refractive index at a multiplicity of locations 

in the medium; 

analytically determining the path of a ray through the multiplicity of locations in 
the medium, for a plurality of rays impinging thereon in different directions. 

36. Apparatus for confocal microscopy including: 

a ray tracer, employing known variations of the refractive index in a three- 
dimensional sample for determining the paths of a multiplicity of rays emerging from at least 
one point in the sample and passing through the sample, thereby determining an aberrated 
wavefront for each the point; and 

an adaptive optics controller utilizing the aberrated v/avefront to control an 
adaptive optical element in a confocal microscope, thereby to correct aberrations resulting 
from the variations in the refractive index. 

37. A method for confocal microscopy including: 

determining variations of the refractive index in a three-dimensional sample; 

determining the paths of a multiplicity of rays emerging from at least one point 
in the sample and passing through the sample, thereby determining an aberrated wavefront for 
each the point; and 

utilizing the aberrated wavefront to control an adaptive optical element in a 
confocal microscope, thereby to correct aberrations resulting from the variations in the 
refractive index. 
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ABSTRACT 

Apparatus for computational adaptive imaging comprises the following: 

an image information acquirer, which provides information relating to the 

refractive characteristics in a three-dimensional imaged volume; 

a ray tracer, which uses the information relating to the refractive characteristics 

to trace a multiplicity of rays from a multiplicity of locations in the three-dimensional imaged 

volume through the three-dimensional imaged volume, thereby providing a location dependent 

point spread function; and 

a deconvolve^ which uses the location dependent point spread function, to 

provide an output image corrected for distortions due to variations in the refractive 

characteristics in the three-dimensional imaged volume. 
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ABSTRACT;] 


Differential Interference Contrast microscopy (DIC) is the 
preferred imaging mode for studying live unstained cells and embryos. This 
is mainly attributed to the optical sectioning capability which makes DIC 
suitable for three-dimensional microscopy. However, DIC images are not 
convenient for standard computerized image interpretation. We present 
here a processing algorithm that converts DIC images into a form which i s 
much more amenable for such analysis. The algorithm is based on 
computational inversion of the directional gradient performed optically by 
DIC, namely path integration (Line Integrated DIC, or LID). LID images 
relate to the refractive index of the specimen, are positively defined and 
therefore many powerful algorithms that were developed for fluorescent 
images can be applied to them. The method is demonstrated here for 
identifying and tracking nuclei in developing zebrafish embryos, and for 
reconstructing the shape of live leukocytes. 
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INTRODUCTION 

Digital image processing has been widely applied for structural 
and quantitative analyses of microscopic images from biological 
specimens. The use of digital imaging for three-dimensional dynamic 
recording of live samples requires methods for automatic processing of 
large data sets. Ideally, computerized analysis should perform 
morphometric and spectroscopic measurements, recognize structural 
features, follow them in time to track changes like those occurring during 
morphogenesis of cells, tissues and whole "organisms and quantify them. 
When the objects of interest can produce high-contrast images, binary 
thresholding help interpretation based on segmentation algorithms like 
skeletonization (e.g. Berns and Berns, 1982) or domain isolation (by sorting 
connected pixels, Press et al., 1986, Benveniste et al., 1989, Kam et al., 
1991). Algorithms for continuous adaptive thresholding were also 
developed for two- and three-dimensional microscopy (Malpica et al., 
1997; Kam et al., 1992; Zamir et al., submitted). Segmentation is simple 
and powerful when applied to fluorescence microscopy, due to the 
specificity of the labeling, the low background and the availability of 
optical sectioning methods for thick specimens. However, these approaches 
are not applicable to the most powerful microscopic mode of imaging for 
live unstained samples, namely differential interference contrast 
microscopy (DIC or Nomarski). DIC car/ dearly record 3D features in 
tissues of many cell layers. Depending on the optics, whole cell 
organization and subcellular organelles can be visualized. The contrast in 
DIC images reflects the spatial directional gradient along the shear angle 
of the refractive index, and is contributed by a thin focal section (typically 
a fuction' nf mirr"m"t"r hi g h mimprir.ai apartures^. DIC can generate 
sufficient contrast even for sub-resolution objects (like microtubules, 
Walker et al., 1988, 1989; and nanometer-displacements of spheres, Gelles 
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et al., 1988). 'When dia 


ing larger objects like cell 


• 


|IC images outline 


the borders between regions of different optical densities such as the 
nucleus and cytoplasmic membranes. However, being a directional gradient, 
DIC images have no contrast perpendicular to the shear angle. This property 


excludes the use of skeletonization and contour closure for defining the 
extent of cellular structures. Recently, procedures that aid visualization 
and interpretation of DIC images were developed like depth projection of 
DIC focal series (Hausler and Korner, 1987; Schormann and Jovin, 1992), 
and inversion of the phase problem based on rotation of shear angle (Preza 
et al., 1997), but these works do not offer a practical solution to alleviate 
the difficulties in applying automated analyses to these types of images. 


convert DIC images to a form applicable for commonly used image 
processing algorithms. We show that line integration of DIC images (LID) 
along the optical shear angle (usually 45°) reconstructs a good 
approximation to the refractive index of the specimen, yielding a 
positively-defined image. As scattering and depolarization from out-of- 
focus regions partially destroy ideal DIC beam conditioning, these 
contributions must be taken into account when applying this method to 
thick, inhomogeneous specimens. An improved guess for the refractive 
index in thick samples can be iteratively computed using constrained 
deconvolution. This method is stable for the positively-defined LID images, 
but can not be applied to the original DIC images, which display both 
positive and negative values (about a positive bias level). 

We demonstrate the application of this procedure for 3D DIC 
imaging of developing zebrafish embryos for identifying and tracking their 
n u c j e i ,-an d— f or— do t c r-roi n at io n of l eukocytes shape. — 



In this paper we demonstrate that a simple transformation can 
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L^^ND METHODS 


MATERIALS^AND METHODS 
Oil droplet calibration slide 

The tip of a Pasteur pipette was filled with immersion oil and 
inserted in a strong, stream of clean air directed towards a coverslide. The 
sprayed drops density and size could be roughly controlled by coverslide 
distance from the tip and air stream strength. The coverslides were placed 
on a drop of water over a glass slide with spacers made of cut #0 
coverslides. Images were acquired on Zeiss Axioskop microscope 
(Oberkochen, Germany) by a scientific grade slow scan high resolution 
(1024X1024 pixels) cooled CCD (Photometries, AZ) using Plan Neofluar 
X100/1.3 objective (Zeiss #440483) and its adjustable DIC prism 
(#444480) 

Zebra fish imaging 

Fertilized eggs were dechorionated and mounted in a sealed 
chamber between two coverslides. The microscope stage was heated to 
28.5°C by air flow incubator. Development proceeded in this chamber 
normally till hatching. Images were collected using the cooled CCD with 
Achroplan X40/.75 water immersion 1.5mm working distance objective 
(Zeiss #440090 and DIC prism #444451). Final magnification on the CCD 
could be adjusted in order to capture the whole embryo in the imaged field 
(600jj.m diameter for 1000 pixels is somewhat coarser then required by 
optimal Shannon/Niquist sampling to pres'erve the optical resolution, but 
sufficient for resolving the nuclei, about 6|um in diameter). Alternatively, 
we zoomed into regions of interest like the epibolizing germ ring as shown 
in this paper. Three-dimensional series were acquired with 2-3jj.m focal 
steps, thus every nucleus was well resolved in two consequtive optical 


' Leukocyte ,^j^ging 

Images were E^qui red in Sedat's laboratory by Orion Weiner. A 
finger-pricked blood droplet was allowed to coagulate on a glass slide, the 
blood clot washed, and the slide was mounted for imaging by Zeiss Axiomat 
X100/1.3 objective with DIC optics. Fiber-optical transillumination 
scrambling was applied, with on : line pixel-by-pixel correction for 
nonuniform field illumination and variations in CCD sensitivity. 

Image processing and visualization 

Image processing and visualization programs were developed for 
use with the "Priism" software package which runs on Silicon Graphics 
workstations (available from Applied Precision, Issaqah, WA 98027-8929). 
The program can display and process movies containing hundreds of high- 
resolution multidimensional images. Processing routines use automated 
and interactive image interpretation algorithms, which may be used on- 
line or as batch processes. 

RESULTS 

The LID algorithm 

The stages in LID processing of the acquired DIC images are 
summarized schematically in the flow chart (Table 1). Following is the 
explanation of the mathematical operations involved. 

DIC image contrast is generated" by interference between two, 
slightly displaced beam cones focused to a point in the specimen. Positive 
and negative contributions correspond to differences in the accumulated 
optical paths between these two beams. The dominant contribution to these 
differences occurs near the plane of focus, where the beam displacement i s 
i rab l.o to tho ' ox tent-of-tho-dif fraction limited region traversed by the 
converging beam at focus. Out-of-focus differences are small, since both 
defocused beam cones average practically identical optical paths over 
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mostly overlapping regions (neglecting the smali^contribution of the 
displaced boundary). As the displacement is designed to be about half the 
focused beam extent, DIC mode preserve the objective resolution. 

Given the . bright-field three-dimensional point-spread function 
(3D PSF) for an objective, P(x,y,z)=P(r,z), the DiC-PSF, D(x,y,z)=D(r J z), can 
be approximated by a difference between two bright-field PSFs displaced 
by a "shear vector" Ar=(Ax,Ay), which direction ("shear angle", usually 45°) 
is defined by ^the aligned polarizer/analyzer and Wollaston prism 
orientations and its length is set by the Wollaston prism shear (Cogswell 
and Sheppard, 1992): 

( 1 ) DCr I z)=PCr+AL/2 ) z)-PU- Ar72,z) 

Identical amplitudes for the two PSFs can be assumed, which is 
sufficiently close to the practical situation. The DIC-PSF shows in-focus a 
displaced difference between two diffraction-limited bright-field PSF 
peaks. Fig 1A illustrates the features of the in-focus DIC-PSF. It can be 
evaluated theoretically (e.g. evaluating P(x»z) according to Stokseth, 1969 
and using Eq.1 to obtain D(r,z), Galbraith, 1982), or measured from the 
images of small beads (P(r.z) from fluorescent beads, or D(r,z) directly in 
DIC mode, Gelles et al., 1988). 

The amplitude of the out-of-focus DIC-PSF vanishes as the extent 
of the defocusing beam (out-of-focus section of the beam cone) becomes 
comparable to Ar, Although the maximal' amplitude of the image of a 
defocused fluorescent point source in wide field microscopy also vanishes, 
the integrated intensity remains roughly constant even far away from the 
focus. On the other hand the out-of-focus integrated intensity for DIC 
vanishes very fast, which is the origin of its optical sectioning property. 

Whon n thrnn dimensional gpoHmgn with refractive index 


0(x,y,z)sO(r,z) is imaged using DIC optics, the recorded image at focal 
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plane z] l(x,y l z)sl(£ l z)afcull be given by the three-dim^^^onal convolution, 

® 3 , with the DIC-PSF,^^,z): 

( 2 ) I(lz) = 0(r,z) ® 3 DCf,z) + B . 

B is a bias level generating, on the average, a gray background intensity 
over which positive and negative features are added, reflecting refractive 
index variations. Due to the optical sectioning property discussed above, 
the 3D convolution can be well approximated by decomposition, for every 
separate z, into section-by-section 2D convolutions, ® 2 , with the in-focus 
PSF, D(r,z=0): 

( 3 ) I(lz) = 0(r,z) ® 2 D(£0) + B. 

The result of the above convolution for the simulated two-dimensional disk 
in Fig. 1B is presented in Fig 1C. The shadow-like appearance of the disk 
mimics DIC images of nuclei. 

Using Eq. 1, the two-dimensional DIC convolution can be expressed 
as two consecutive "optical operations" on the refractive index: 
convolution with the bright field PSF, P(x,y,z=0)=P(i,0), and then sheared 
difference: 

( 4 ) J(r,z) = OUz) ® 2 P(lO) . 

( 5 ) I(lz) = J(r+At/2 f z)-J(r- Ar/2,z) + B ~ 3J(r,z)/a£ • Ar + B 
I(lz), the measured DIC image, is closely approximated by a directional 
gradient (along the shear angle) of J(lz). As Ar. is comparable to the 
resolution of the image, the finite difference in Eq. 5 and the analytical 
gradient (the limit for Ar.->0) are infinitesimally close. The inverse 
operation, namely line-integration, will retrieve J(r,z) from l(r,z): 

( 6 ) J(£ J z) = J [ I(t+sAlz) - B ] ds . 

-oo 

where oo means in practice that integration starts from the extreme 
borders of the image. This inverse operation for the 2D image consist of 
actually independent 1D integrations, carried for each line and covering the 
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"IDis the LID image resulting from^L 


whole image. Fig 1Dis the LID image resulting from^uch integration along 
45° lines for the DIC image at Fig 1C. It reconstructs the disk-like 
refractive index distribution. Note the disk edges are smeared due to the 
finite resolution of the bright-field PSF. 

As seen in Fig. 1C, the "shaddow" casted by the disk is strongest 
along the shear angle and diminishes perpendicular to it. Eq. 5 implies a 
decreas proportional to the dot product between the shear and the vector 
vertical to the disk edge, giving a cos(p) factor, (p angle with the shear). 
Nevertheless, integration along all parallel lines crossing the shaded disk 
yield the same value inside the disk in Fig. 1D. The reason is that the line 
integration along the shear angle is crossing the disk edges in a path 
longer by a factor of 1/cos(p), compensating for the lower shaddow. 
Although this result holds true analytically even for the direction 
perpendicular to the shear, in practice integration tangential to high 
contrast features may cause some artifacts (see weak streaks extended at 
45° from both sides of droplets in Fig. 2B). 

DIC images of thick specimens, such as embryos, include in 
addition to sharp in-focus gradient-like features, also slow-varying 
components. These result from disruptions of the DIC beam conditioning, 
for example by scatter and depolarization, that occure in the out-of-focus 
regions of the specimen. To eliminate these components we pre-process 
the three-dimensional stack of DIC images by subtracting out-of-focus 
contributions using the approximated nearest neighbor deconvolution 
(Agard and Sedat, 1984): 

(7) I"(LZ) = I(lz) -a*{rCl,z+Az) +1'Cr,z-Az)} 

where I*(lz±az) are the two-dimensional (blurring) convolution of the 

■Sfrfitinnq With frtrilS A 7 ahnup/halnw 7 W jfh th Q ^rrogpnnHing nnt.nf.fnn.ic 

PSF, P(r,±Az): 

(8) r(LZ±Az) = IQ;z±Az) ® 2 P(r,±Az) . 
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Eq. 7 can be consider^jj^n elaboration of "mottle-su^J^tion" that was 
applied in enhanced video microscopy to relatively thin specimens (Allen et 
al., 1981; Inoue and Spring, 1997). For thick specimens like zebrafish 
embryos, nearest . neighbor deconvolution was found to effectively 
eliminate out-of-focus contributions of the yolk cell, a is a scale factor 
which, for fluorescence, assumes optimal value of 0.45 (Agard et al., 
1989). Unlike the idealized fast vanishing of the out-of-fbcus PSF for DIC, 
the experimentally-measured contributions for thick specimens cannot be 
accounted for by an optical model. We therefore tested the value of a 
visually according to the elimination of identified out-of-focus features. 
For both the zebrafish embryo and the leukocyte cells a=0.45 was found 
adequate (Fig. 4A). Smaller or larger values clearly caused under- or over- 
subtraction. For DIC images that present a single focal plane, subtraction 
of low frequency image components (high-pass filtering) is an excellent 
substitute. We employ a fast "box filter" algorithm (Kam et al., 1992) with 
box size larger than a typical size of features of interest in the image. 

Nearest neighbor out-of-focus subtraction creates a visibly "flat" 
background, yet the long integration path in Eq. 6 accumulates random noise 
(proportional to the square root of the number of pixels in the integration), 
and picks also slowly varying image components. We suppress this effect 
by including in the integration an exponential "decay" factor chosen to be 
larger then the size scale of the features of interest (e.g. the nuclei). Since 
DIC-PSF is anti-symmetric with respect to a mirror plane perpendicular to 
Ar, integration in opposite directions are inverse in sign. Taking the 
difference between integration in two opposite paths along the shear angle 
("bidirectional, decay") cancels the asymmetry introduced by the decay, as 

-well— as eliminates the eontributio n-of— t he- b i as l o vol B: 

r° 

(9) J'(r,z) = J {r , a , +sALz)-l ,, (r-sAr J z)}exp(s/s )ds 
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In order to reduce noise, the integration algomnm allows optional 
averaging along direction Ag. perpendicular to the shear vector Al in a strip 
of width ±d that is taken comparable to the resolution: 


Fig. 1E shows the effect of the bidirectional decay on large uniform regions 
of higher or lower refractive index, where LID ; image intensities are 
correctly represented close to boundaries, but tend towards zero further 
away. - - - 


resulted refractive index. The correction is estimated from comparison 
between the acuired DIC image, and convolution of the LID image with the 
DIC PSF. This diminishes the effects of the approximations introduced in 
the "inverse" derivation of LID from DIC (like the decay factor and 
integrative accumulation of noise). The inherently ill-defined 
deconvolution algorithm have been shown to be stabilized by imposing 
constraints like positivity for fluorescent images (Sezan and Stark, 1987; 
Agard et al., 1989; Koshy et al., 1990; Carrington, 1990; Snyder et al., 
1992). However, positivity cannot be applied directly to DIC images that 
contain positive and negative values (about an arbitrary bias). By 
converting the problem to that of solving for the positively-defined 
refractive index, positivity can be imposed, and an improved solution can 
be iteratively approached. In fact for biological specimens, with . a known 
range of refractive indices, even tighter constraints can be imposed. The 
iterative steps for such LID deconvolution are performed for each z section 
as follows (see also flow chart in Table 1): 

1. Initial approx imation to the refractive index, Jo M (r,z) f is cal culated 
from the preprocessed (Eqs. 7,8) DIC image, I m (lz). by line-integration (Eq. 



-d 


We use iterative constrained deconvolution to improve the 
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2. Corrections are th^^pplied iteratively: 

M-L.z), the predicted D1C image, is calculated by convoluting the 
N-1 iteration of the refractive index, JnVC^z), with the in-focus DIC-PSF 
^ (Eq. 3, ignoring the bias, B, since it is eliminated in the bidirectional 
integration). 

aIn(I,z), the difference between InU» z ) ancl the observed 
preprocessed image I m (lz), is LID integrated (Eq. 10), and the resulting 
incremental correction to the refractive index, aJn(I,z). < s added to the 
previous estimate J N .i ,, Cr I z) to yield J N "(r,z). The result is clipped if lower 
then the constraint, and convergence criterion is applied to terminate the 
iterations. The processing time for 512X512 image on Silicon Graphics 
INDY is 15-30 seconds. 

Fig.lF shows deconvoluted LID images after 3,6,9 and 12 
iterations. The "filling" of the inside of the disk can be seen. Thus the 
deconvolution iterations can corrects for the decay effects. Line-to-line 
noise caused by long decay factor is also rectified by the deconvolution. 
Note that the amplitude of high frequency components, evident at the disk 
edges, stay bound due to the stabilization by the constraint. 

Applications 

Fig. 2A is a microscopically acquired DIC image of immersion oil 
droplets in water, showing features similar to the simulated disk in Fig. 1 . 
Fig. 2B is the result of LID processing that converts the shadow-like 
gradient image into an image showing the droplets as bright disks of higher 
refractive index. Fig. 2C is the deconvoluted image. Since the droplets stick 
to. the coverslide as half-domes, larger size implies thicker, oil layer. Still, 
the br i ghtness— trr^her-btB i mage for droplets larger— th an a bout 0.5 jam is= 
independent of their diameter. This is a quantitative confirmation that DIC 
imaging can be separated into independent 2D optical sections, which 
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allows to calibrate itm LID image brightness in terrW^'of local refractive 
index differences. A series of LID images of droplets were acquired with 
incremental adjustment of the DIC prism screw (the displaceable 
^Wollaston prism behind the objective), and the average brightness of the 
disks is plotted as a function of the adjustment (Fig. 2D). The linear 
depehdence of brightness on DIC prism adjustment relates the slope with 
refractive index difference between oil and water. Once calibrated, and 
given the adjustment, the corresponding LID intensity for oil droplets in 
water, J c , is read from Fig. 2D and LID intensity, J, can be mapped to 
refractive index difference, An according to: 
(11) An = (n oi i -n water )*j/j c 

For biological samples, where the borders of the DIC image is the solution, 
the absolute refractive index simply equals n wate r + An . J can be positive 
or negative (i.e. for air bubbles), but for cells is typically in a narrow range 
between n wat er and 1.5 (for chromatin). 

The DIC PSF, needed for the deconvolution, can be measured by 
imaging a small bead. This method is very sensitive to slight defocus, and a 
finely spaced focal series should be analyzed (e.g. 0.05 u.m steps for 
X100/1.3 objective). When the bright field PSF is known (usually from 
images of small fluorescent beads) one can obtain Ar by best-fitting the 
image of a bead or a diamond-scratched line with the two dimensional 
convolution (Eq. 3) between the in-focus, Ar-dependent DIC PSF (Eq. 1) and 
a simulated point or line respectively. Alternatively the DIC image itself 
contains the information about Ar.. The image created by DIC optics is the 
convolution of the refractive index with the objective DIC-PSF. This 
convolution can be carried by multiplication in Fourier space with the 
c ontrast tr a n s f e r f u nction, CTF, w hi c h io tho F o urio r t ranofo r m (FT ) of th o 
PSF. The FT of DIC images are therefore modulated by the CTF. Fig. 3 
displays log magnitude of such FT (or the power spectrum, zero frequency 
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DISCUSSION ^ 

in this work we developed line-integration algorithm (LID) for 
transforming light microscope DIC images into refractive indices. The 
£ "shadow-casting" presentation of nuclei and cells in DIC is transformed 
into positively defined solid bright filled areas. Iterative constrained 
deconvolution can be applied to approach improved refractive index 
estimate for three-dimensional optical sections of thick samples. Unlike 
the unprocessed DIC images, LID images can be presented to computerized 
image interpretation methods that were developed for fluorescent images. 
Such algorithms help analyze large sets of image data. In our application, 
embryonic development involves recording of spatio-temporal information 
which analysis includes thousands of images, occupying Gigabytes of disk 
space. The present work allows automated nuclei identification and cell 
lineage tracking in large domains of a developing embryo, for dynamic 
presentation of morphogenetic "flow patterns" in wild type and mutants 
and for studies of tissue differentiation and fate determination. Previous 
work with similar goals was mainly based on flourescence following 
microinjection of specific DNA labels or of tracer dyes in a few nuclei. 
Recent work used DIC and manual cell tracking. 

From leukocyte LID images automatic 3D shape reconstruction i s 
possible, and movies of motility can be created. This task required in the 
past intensive manual work (Murray et al., "1992; Felder and Kam, 1994). 
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Table I ' 
FLOW CHART FOR THE RECONSTRUCTION OF LID FROM PIC IMAGES : 
Symbols used: 

P(_r,z): The 3D bright field PSF for an objective: 

Theoretical (e.g. Stokseth, 1969) or experimental (fluorescent beads). 

D(_LZ): The DIC PSF: (see for example Fig.lA) 

Eq.1 D(r,z) = P(r+AL/2,z) - P(r-Ar/2,z) 

Ar = shear vector. Obtained by best-fit of images of beads (or 
scratches) to S(r,0) ® 2 D(j,o) J S = simulated point (or line) image. 
Alternatively, fit power spectrum of DIC image to CTF*sin(k*Ar) . 

I(LlZ) ■ The measured DIC image at focal plane Z. (see for example Rg.ic) 
J(r,z) : The corresponding (computed) LID image. (see for example Fig. id.e.f) 
Eq.11 LID calibration in terms of refractive index: An = (n 0 j| -n wa ter)*J/Jc 

Nearest Neighbor deconvolution for focal series of DIC images. 
Eq.7 r(LZ) = l(r,z) -a *{r(r,z+Az) + I'(lz-Az)} 

Eq.8 r(LZ±AZ) = ICr,Z±Az) ® 2 PCr,±Az) 

Az is the focal step, a = 0.45. 

Can be replaced for a single section image by high pass filtering. 

Iterative, section-by-section reconstitution (for every z): 

Initial guess for the LID image by line integration: (see Fig.iE) 

o. d 

Eq.10 Jo'^JLz) = J J {l"(r+sAr+ tAq f z]--["(r- sAr + tAq ,z)}exp(s/s Jdsdt 

- oo - (j 

Integration range, d, is comparable to the resolution. Ag. _lAt_. 

LID deconvolution iterations (N=1,2, ...) 

Eq.3 l N (r,z) = J N .r(l,z) ® 2 D(lz=0) 

AInU.z) =r(Lz) - ! N (_r,z) 
o d 

r q m A l^r^)- j [ ^A , l f| (r i o Ar j, M q | 7 )-Al M fr- c^fa, tAq^) j pYp(-ft/a o )dSClt 

-oo -d 

JN"(l.z)=JN-i"(r.z)+AJN(£,z) (Jn"(I.z) ^ low constraint) (see Rq.if) 
check convergence: \ {AJNXr.z)} 2 <2f/ J {Jn+1 '(l.z)} 2 dr « 1 
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FIGURES 



Fig. 1 Illustrative simulation of DIC and LID images. For the purpose of 
£ quantitative presentation, intensity line-scanned profiles are shown. 

Fig. 1A Light microscope in-focus point-spread function (PSF) in DIC 
mode, calculated as a difference between two, bright-field point-spread 
functions, displaced along 45° shear angle by half their width. 
Experimentally measured (e.g. small fluorescent beads) or theoretical 
objective PSF (Stokseth, 1969) can be used. Here two-Gaussian-difference 
is displayed for simplicity. 
Fig. 1B Simulated disk. 

Fig. 1C DIC image calculated by convolution of the disk of Fig. 1B with 
the PSF in Fig. 1A. The shadow-casting appearance is typical for DIC 
images of nuclei. 

Fig. 1D LID image, calculated by 45° line integration of Fig. 1C, which 
retrieves the original disk in Fig. 1B blurred due to the in-focus bright- 
field PSF finite resolution. 

Fig. 1E LID image of Fig. 1C with bidirectional decay. In real images it 
suppresses random noise and subtracts the bias, but also reduces the LID 
intensity within large areas of higher refractive index, like inside the disk. 
Fig. 1F LID deconvolution, "filling" the reduced intensity inside the 
disk. Subimages 1-4 correspond to the result following 3,6,9 and 12 
iterations. Note that the amplitude of the high-frequency oscillations at 
the disk edge stay bound due to the constraint. 

Fig. 2A DIC image of oil droplets sprayed on a coverslide in water. 
Sual tf bai= l u |Iiu. 

Fig. 2B LID image corresponding to Fig. 2A, depicting the oil droplets as 
bright areas. 
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Fig. 2C Deconvolved LID image showing increa^S ''contrast and reduced 
"decay" effects. 

Fig. 2D Average droplet brightness for a series of LID images obtained 
as a function of DIC prism adjustment. The dependence is fitted by a line, 
crossing zero at the middle of the adjustment range, with positive 
brightness for adjustment to one side, and negative brightness for the 
other side due to inversion of the DIC shadow-casting effect. The fitted 
line allows to calibrate specimen refractive index differences in terms of 
LID image brightness: DIC prism adjustment (set by known angle of turn of 
the prism screw) gives the corresponding LID intensity for oil-water 
difference, J c , (dashed line) for insertion in Eq. 11. AU: arbitrary unit 
(corresponds to 90° incremental turn of the DIC prism screw). 

Fig. 3 Log power spectrum of a DIC image. Zero frequency is at the center of 
this image. The splitted peaks (see intensity line scan profile) are the 
result of the single PSF peak modulated by the plane wave, sin(k*Ar), due 
to the Wollaston prism shear. Scale bar= 5 u.m - 1 . 

Fig. 4A A focal section within a live zebrafish embryo during epiboly. 
Nuclei of cells near the spreading front (moving from left to right) are 
displayed. The DIC image also strongly contrasts cell outlines, resulting 
from cell-hight accumulated optical path through intracellular fluids, and 
contain strong diffused contribution accumulated from refraction and 
scatter in the long optical path within the yolk. Nearest neighbor 
subtraction of out-of-focus contributions displays here reduced yolk and 
lower intercellular shadows. Scale bar= 50 u.m. 

Fin 4R Iterativa rWnnunh ■ting tn „ pic PS F i n ing I IP "rre described in 
the text, showing the nuclei as white circles. Contributions of the 
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intracellular fluids are|^j high, but can be morpholo^J^lly distinguished 
from the nuclei. 

Fig. 5A DIC image focused within a leukocyte, displaying convoluted 

lamellapodia structures. 

Fig. 5B 1 LID of the image in 5A. 

Fig. 5C Automated outline contouring of the leukocyte shape cross 
section by thresholding the LID image in Fig. 5B. Scale bar= 5 (im. 
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